Add BrowserStack SDK + Playwright behave sample by Jimesh-browserstack · Pull Request #1 · browserstack/behave-playwright-browserstack

Jimesh-browserstack · 2026-05-04T11:53:27Z

Summary

Customer-facing starting point for running behave tests on BrowserStack via Playwright Python and the BrowserStack Python SDK. Mirrors the shape of browserstack/cucumber-ruby-browserstack (browserstack-sdk-selenium-4 branch) but adapted for behave + Playwright.

What customers get

.
├── browserstack.yml                  # 3-platform parallel default (chromium/firefox/webkit)
├── config/
│   ├── browserstack.single.yml       # 1 platform
│   └── browserstack.local.yml        # 1 platform + browserstackLocal: true
├── requirements.txt                  # SDK + behave + playwright==1.49.0 (pinned)
├── Makefile                          # parallel / single / local / install
└── features/
    ├── sample.feature                # bstackdemo add-to-cart
    ├── local.feature                 # localhost via tunnel
    ├── local-html/index.html         # static page served on :45454 by `make local`
    ├── environment.py                # sync_playwright + chromium.launch + page wiring
    └── steps/
        ├── sample_steps.py
        └── local_steps.py

Three customer-facing entry points:

make parallel — runs across the three Playwright engine families (chromium / firefox / webkit) in parallel on BrowserStack
make single — one chromium platform
make local — one chromium platform + BrowserStack Local tunnel against a local HTTP server the Makefile spins up on :45454

Design notes

Customer code is browser-agnostic. features/environment.py calls chromium.launch() unconditionally; the SDK monkeypatches Playwright at runtime to route the launch to the per-platform browser configured in browserstack.yml. No chromium.connect(wss_url) plumbing needed in customer code (this differs from the Java SDK behavior).
Config swapping mirrors the Ruby reference. The SDK only reads ./browserstack.yml, so single and local Make targets copy in alternates from config/, run, and restore the parallel default afterwards (even on failure).
Local mode self-tests. Rather than running bstackdemo through an unused tunnel, make local serves a small static page on localhost:45454 and routes a localhost-targeting scenario through the BS Local tunnel that the SDK starts/stops automatically.
Playwright is pinned to ==1.49.0. PW >=1.50 trips an SDK monkeypatch arg that no longer matches (unexpected keyword argument 'artifactsDir'). Comment in requirements.txt explains.
CI workflow follows the cucumber-java-playwright-browserstack pattern: workflow_dispatch only with a commit_sha input, posts check status back to that SHA. No push / pull_request triggers, so BrowserStack minutes aren't burned on every commit.

Local verification (already complete)

Ran each Make target end-to-end against BrowserStack from a fresh clone with creds in env vars:

Mode	Build	Sessions	Result
`make parallel`	3ea91fa0…	3 (chrome 147 / playwright-firefox 148 / playwright-webkit 18.2)	all 3 passed
`make single`	e64fd864…	1 (chrome)	passed
`make local`	d47aa127…	1 (chrome via BS Local tunnel → localhost:45454)	passed

Cleanup checks on every run: browserstack.yml restored to 3-platform default after single and local, no leftover .bak file, no orphan python3 -m http.server on :45454.

CI verification (pending post-merge)

workflow_dispatch workflows only become dispatchable from the Actions UI once they exist on the default branch — so CI can't be validated against this branch before merge. After merge, dispatch the workflow against main HEAD; if BrowserStack auth fails it means the org-level Actions secrets (BROWSERSTACK_USERNAME / BROWSERSTACK_ACCESS_KEY) aren't wired through to this repo and need to be added by an Actions admin (same setup as cucumber-java-playwright-browserstack).

Test plan

make parallel runs all 3 platforms in parallel — see Build URL above
make single runs one chromium session
make local starts BS Local tunnel, serves localhost page, scenario passes
browserstack.yml is restored after single / local, no .bak leftover
No orphan python3 -m http.server on :45454 after make local
workflow_dispatch runs green post-merge (pending — see "CI verification" above)

🤖 Generated with Claude Code

Provides customers with a working starting point for running behave tests on BrowserStack via Playwright Python and the BrowserStack Python SDK. Mirrors the shape of browserstack/cucumber-ruby-browserstack (browserstack-sdk-selenium-4 branch) but adapted for behave + Playwright: - One browserstack.yml at root declares 3 platforms covering all 3 Playwright browser engines (chromium / firefox / webkit). The SDK monkeypatches Playwright launches and routes to the per-platform browser, so customer code uses chromium.launch() unchanged across all three. - Alternate configs under config/ for single-platform and BrowserStack Local modes; Makefile swaps them in around the SDK call and restores the parallel default afterwards (even on failure). - Local mode self-tests: serves a small static page on :45454 from features/local-html/ and routes a localhost scenario through the BS Local tunnel that the SDK starts/stops automatically. - GitHub Actions workflow uses workflow_dispatch + commit_sha + check-status reporting, mirroring the cucumber-java-playwright-browserstack pattern. - Playwright pinned to ==1.49.0 because PW >=1.50 trips an SDK monkeypatch arg that no longer matches (`unexpected keyword argument 'artifactsDir'`). Verified end-to-end against BrowserStack: - make parallel: 3 sessions (chrome/pw-firefox/pw-webkit), all passed - make single: 1 chrome session, passed - make local: 1 session via Local tunnel against localhost:45454, passed Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

karanshah-browserstack · 2026-05-07T06:26:08Z

+# the launch keyword args in a way the current SDK doesn't yet handle
+# (`unexpected keyword argument 'artifactsDir'`). Keep at 1.49.0 until the
+# SDK release notes call out a higher upper bound.
+playwright==1.49.0


Why it is not latest and pointed to specific version?

There seems to be an issue in SDK for PW > 1.50.
Ticket raised here https://browserstack.atlassian.net/browse/SDK-5954

Jimesh-browserstack · 2026-05-07T06:49:52Z

Thanks for the review @karanshah-browserstack — pushed 0f337d2 which addresses threads #1, #2, and #4 directly. Thread #3 (README format) still needs your input — see the bottom of this comment.

Re: `config/browserstack.single.yml` — `buildIdentifier: '#${BUILD_NUMBER}'` ✅ resolved

Why is this needed?

Dropped. The whole config/ directory is gone, and buildIdentifier is removed from the remaining browserstack.yml — agree it was pure dashboard UX, not required for a sample. If we miss it, easy to add back.

Re: `Makefile` ✅ resolved

Why is makefile needed?

Dropped. We followed your instinct to look at this honestly — the Makefile was a Python equivalent of cucumber-ruby-browserstack's Rakefile, but the only thing it really earned was the per-mode config swapping, and we've removed all the modes except the default. Now there's one command:

browserstack-sdk behave features/sample.feature

Run directly, both locally and in CI (see .github/workflows/build.yml).

Local-tunnel mode is documented as a one-line config flip (browserstackLocal: true in browserstack.yml) rather than a separate config file + feature + Make target.

Re: `requirements.txt` — `playwright==1.49.0`

Why it is not latest and pointed to specific version?

Playwright >=1.50 triggers an immediate failure with the current Python SDK at session start:

TypeError: bstack1l111l11l1_opy_() got an unexpected keyword argument 'artifactsDir'

The SDK monkeypatches Playwright's BrowserType.launch() and passes a kwarg that newer Playwright launchers don't accept. PW 1.49.0 is the last version verified working against browserstack-sdk==1.46.0. Will move the pin once the SDK release notes call out a higher upper bound — if there's an existing SDK-team tracking ticket I'll link it from the comment in requirements.txt.

Re: `README.md` ⏸ awaiting input

This won't look good, let's keep it consistent with existing sample repos?

I modeled this PR on cucumber-ruby-browserstack (master) — same SDK family, similarly short README. After today's simplification the structure now matches it almost 1:1 (one yml, one feature, one command).

Existing samples diverge in shape though, so want to confirm before any prose rewrite. Which template should the README mirror?

cucumber-ruby-browserstack — short README: Setup / Running / Further Reading (current shape)
cucumber-java-playwright-browserstack — long README: BrowserStack logo + Prerequisites / Quick start / Setting credentials / Localhost / Project layout / Customizing / Adopting / Troubleshooting / Notes
testng-browserstack — if there's a different canonical we're standardizing on

Happy to rewrite once we pick.

karanshah-browserstack · 2026-05-07T07:03:43Z

+consoleLogs: errors
+
+# Identifier so BrowserStack can tag the sample source — please leave as-is.
+source: behave-playwright-sdk:sample-master:v1.0


behave-playwright-browserstack

Fixed in fa69904 — was behave-playwright-sdk:sample-master:v1.0 (my made-up -sdk prefix). Updated to behave-playwright:sample-master:v1.0 matching the existing convention: cucumber-ruby uses cucumber-ruby:sample-master:v1.1, cucumber-java-playwright uses cucumber-java-playwright:sample-master:v1.0 — pattern is <framework>-<library>:sample-master:v<N>. Happy to coordinate the version with the ASI team before merge if you'd like.

Karan's review (#1, #2, #4) pushed back on the per-mode config files and the Makefile-as-profile-switcher. Aligning with the canonical cucumber-ruby-browserstack shape (one feature run unchanged): - Drop Makefile and config/{single,local}.yml — the Python SDK reads only ./browserstack.yml so the swap was the workaround; aligning instead means one yml, one command. - Drop features/local.feature, local-html/, and local_steps.py. Customers needing the Local tunnel flip browserstackLocal: true and rerun the same command (documented in README). - Drop buildIdentifier from browserstack.yml — pure dashboard UX, not required, and Karan flagged it as noise. - CI workflow runs browserstack-sdk behave directly (no make targets). - README pruned to single-mode setup + Running + Local-toggle note.

Was `behave-playwright-sdk:...` — my made-up `-sdk` prefix that doesn't match any real convention. Other samples use `<framework>-<library>:sample-master:v<N>` (cucumber-ruby, cucumber-java-playwright), so the analog here is `behave-playwright:sample-master:v1.0`. Addresses #1 (comment)

Was `playwright==1.49.0` pinned to dodge the SDK monkeypatch's `artifactsDir` kwarg mismatch on PW >=1.50. Customers get the latest Playwright now; if the SDK still rejects the new kwarg they'll see `TypeError: ... unexpected keyword argument 'artifactsDir'` until BS ships an updated SDK. Resolves #1 (comment)

karanshah-browserstack · 2026-05-07T10:52:50Z

+consoleLogs: errors
+
+# Identifier so BrowserStack can tag the sample source — please leave as-is.
+source: behave-playwright:sample-master:v1.0


behave-playwright-browserstack

behave-playwright-browserstack:sample-main:v1.0

- requirements.txt: pin to 1.49.0 (avoid drift; matches what we test). - browserstack.yml: source -> behave-playwright-browserstack:sample-main:v1.0 to reflect the actual repo and default branch. Verified end-to-end via SDK: 3/3 sessions passed (chromium / firefox / webkit). Build: 9997a825ccaa7a0eef53e2aaf297763c0d3e7b26

xxshubhamxx · 2026-05-11T05:41:25Z

@@ -0,0 +1,59 @@
+# =============================


Please ensure that the structure is similar to: https://github.com/browserstack/junit-browserstack/blob/master/junit-5/browserstack.yml
A couple of comments are missing.
buildIdentifier key is missing as well.

Addresses xxshubhamxx review: match structure of junit-browserstack/junit-5/browserstack.yml, add missing comments, add buildIdentifier key.

Jimesh-browserstack · 2026-05-12T06:00:15Z

@xxshubhamxx — addressed in 943aefd.

Please ensure that the structure is similar to: https://github.com/browserstack/junit-browserstack/blob/master/junit-5/browserstack.yml
A couple of comments are missing.
buildIdentifier key is missing as well.

Restructured browserstack.yml to mirror the junit-5/browserstack.yml layout 1:1:

BrowserStack Reporting section — added the inline explanatory comments for projectName, buildName, and framework exactly as in junit-5.
buildIdentifier — added with the full comment block (${BUILD_NUMBER} / ${DATE_TIME} expression options + link to the organize-tests docs).
Platforms section — added the "Entire list available here" link to the list-of-browsers-and-platforms page. Kept a short Playwright-specific addendum explaining that chromium.launch() in customer code is routed by the SDK to the per-platform browser, since that is the unusual part of this sample vs. junit.
Parallels per Platform — split out into its own section with the Example 1 / Example 2 narrative from junit-5.
BrowserStack Local — expanded the section header, added the Local docs link, and added the commented browserStackLocalOptions block (localIdentifier, forceLocal, link to manage-incoming-connections).
Debugging features — added the inline <boolean> / <string> annotations and the "Available options are disable, errors, warnings, info, verbose" note after consoleLogs.

Behave-specific values kept: framework: behave, source: behave-playwright-browserstack:sample-main:v1.0, three Playwright engine platforms. Default for browserstackLocal left as false since the README documents flipping it on as the path for private hosts.

PTAL.

Jimesh-browserstack · 2026-05-12T06:15:03Z

Verified 943aefd end-to-end against BrowserStack — fresh clone of add-sdk-playwright-sample, fresh venv (Python 3.10, matches .github/workflows/build.yml), pip install -r requirements.txt, playwright install chromium, then browserstack-sdk behave features/sample.feature.

Build: b21d7d574ec0f4f2579ee81440f0e27468cb354f

REST API (GET /automate/builds/<id>/sessions.json) returns all 3 sessions with status: passed:

Platform	Browser	Build name (returned)	Status
Windows 11	chrome 148.0	`behave-playwright-sdk-build-1 #12`	passed
Windows 11	playwright-firefox 148.0	`behave-playwright-sdk-build-1 #12`	passed
OS X Sonoma	playwright-webkit 18.2	`behave-playwright-sdk-build-1 #12`	passed

The #12 suffix on every build name confirms the new buildIdentifier: '#${BUILD_NUMBER}' is being interpreted by the SDK (not just passed through as a literal). All 3 Playwright engine families (chromium / firefox / webkit) covered, ready for re-review.

xxshubhamxx · 2026-05-12T08:09:57Z

Hi @Jimesh-browserstack
Can we add a sample local test as well?
I can see that we have 2 sample tests in all the other repos.
Eg:

https://github.com/browserstack/junit-browserstack/tree/master/junit-5/src/test/java/tests
https://github.com/browserstack/cucumber-java-browserstack/tree/master/src/test/resources/features (Sample repo for BDD Framework)

Mirrors the 2-sample pattern from cucumber-java-browserstack and junit-browserstack so customers get one public-site sample plus one BrowserStack Local sample. - features/local.feature — navigates http://bs-local.com:45454/ and asserts the page title contains "BrowserStack Local". - features/steps/local_steps.py — two steps. - features/local-html/index.html — title-matching page; start with `python3 -m http.server 45454 --directory features/local-html` before running the local feature. - browserstack.yml — browserstackLocal: true (matches cucumber-java-browserstack canonical so the SDK starts and stops the tunnel for every run). - README.md — repo layout updated; Running section split into "Sample test" and "Local test" subsections with explicit local-server start command. Live-verified end-to-end: build 03f2c770c3b15871829075c20f45ad4a826b56cf returns status=passed for all 6 sessions (2 scenarios x 3 Playwright engines: chromium 148, playwright-firefox 148, playwright-webkit 18.2).

Jimesh-browserstack · 2026-05-13T06:56:46Z

@xxshubhamxx — added in 9f41152.

Can we add a sample local test as well?

Mirrored the 2-sample pattern from your BDD reference (cucumber-java-browserstack):

features/sample.feature — existing public-site sample (bstackdemo.com add-to-cart).
features/local.feature — new: navigates http://bs-local.com:45454/ through the BrowserStack Local tunnel and asserts the page title contains BrowserStack Local. Matches the Local.feature assertion in cucumber-java-browserstack/src/test/resources/features/localtest/Local.feature 1:1.
features/steps/local_steps.py — two new steps.
features/local-html/index.html — minimal page titled "BrowserStack Local Test Page" so the sample is runnable out of the box.
browserstack.yml — browserstackLocal: true (matches cucumber-java-browserstack's browserstack.yml; SDK starts and stops the tunnel for every run).
README.md — repo layout updated; the Running section is now split into two subsections (Sample test / Local test) with the explicit python3 -m http.server 45454 --directory features/local-html start command for the local test, then browserstack-sdk behave features/local.feature in a second terminal.

Live verification

Fresh clone of add-sdk-playwright-sample, fresh venv (Python 3.10, matches .github/workflows/build.yml), pip install -r requirements.txt, playwright install chromium. Started the local server with python3 -m http.server 45454 --directory features/local-html, then browserstack-sdk behave features/ to drive both features.

Build: 03f2c770c3b15871829075c20f45ad4a826b56cf

REST API (GET /automate/builds/<id>/sessions.json) returns all 6 sessions with status: passed:

Scenario	Platform	Browser	Status
sample.feature — add product to cart	Windows 11	chrome 148.0	passed
sample.feature — add product to cart	Windows 11	playwright-firefox 148.0	passed
sample.feature — add product to cart	OS X Sonoma	playwright-webkit 18.2	passed
local.feature — title contains "BrowserStack Local"	Windows 11	chrome 148.0	passed
local.feature — title contains "BrowserStack Local"	Windows 11	playwright-firefox 148.0	passed
local.feature — title contains "BrowserStack Local"	OS X Sonoma	playwright-webkit 18.2	passed

Both scenarios × 3 Playwright engines (chromium / firefox / webkit) all green.

@karanshah-browserstack — flagging FYI: this re-adds features/local-html/index.html, which I'd dropped earlier per your simplification feedback. Re-added in the much simpler form (no Makefile, no config swap) to match Shubham's reference repos and so the local sample is runnable without extra setup. Happy to drop it again if you prefer the "customer brings their own server" pattern that cucumber-java-browserstack itself uses — just say the word.

PTAL.

Jimesh-browserstack requested a review from a team as a code owner May 4, 2026 11:53

karanshah-browserstack requested changes May 7, 2026

View reviewed changes

karanshah-browserstack reviewed May 7, 2026

View reviewed changes

Comment thread features/local-html/index.html

Jimesh-browserstack added 3 commits May 7, 2026 12:45

Jimesh-browserstack mentioned this pull request May 7, 2026

Add BrowserStack SDK + Playwright xUnit/Reqnroll sample browserstack/xunit-reqnroll-playwright-browserstack#1

Merged

6 tasks

karanshah-browserstack requested changes May 7, 2026

View reviewed changes

karanshah-browserstack previously approved these changes May 7, 2026

View reviewed changes

xxshubhamxx reviewed May 12, 2026

View reviewed changes

Restructure browserstack.yml to match junit-5 reference layout

943aefd

Addresses xxshubhamxx review: match structure of junit-browserstack/junit-5/browserstack.yml, add missing comments, add buildIdentifier key.

Jimesh-browserstack dismissed karanshah-browserstack’s stale review via 943aefd May 12, 2026 05:58

xxshubhamxx approved these changes May 13, 2026

View reviewed changes

karanshah-browserstack approved these changes May 13, 2026

View reviewed changes

karanshah-browserstack merged commit 46881c0 into main May 13, 2026
5 checks passed

karanshah-browserstack deleted the add-sdk-playwright-sample branch May 13, 2026 10:31

Conversation

Jimesh-browserstack commented May 4, 2026

Summary

What customers get

Design notes

Local verification (already complete)

CI verification (pending post-merge)

Test plan

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Jimesh-browserstack commented May 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Re: config/browserstack.single.yml — buildIdentifier: '#${BUILD_NUMBER}' ✅ resolved

Re: Makefile ✅ resolved

Re: requirements.txt — playwright==1.49.0

Re: README.md ⏸ awaiting input

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Jimesh-browserstack commented May 12, 2026

Uh oh!

Jimesh-browserstack commented May 12, 2026

Uh oh!

xxshubhamxx commented May 12, 2026

Uh oh!

Jimesh-browserstack commented May 13, 2026

Live verification

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Jimesh-browserstack commented May 7, 2026 •

edited

Loading

Re: `config/browserstack.single.yml` — `buildIdentifier: '#${BUILD_NUMBER}'` ✅ resolved

Re: `Makefile` ✅ resolved

Re: `requirements.txt` — `playwright==1.49.0`

Re: `README.md` ⏸ awaiting input